#3158 - [Jib core] Tar archives with same contents are not reproducible #3159

davidtron · 2021-03-20T21:19:58Z

Sets the mod time on TarArchiveEntry to DEFAULT_MODIFICATION_TIME (Epoch second 1) in order to create reproducible Tar images. Associated unit test creates 2 tars output streams from the same input 2 seconds apart to show the failure if we don't explicitly set the TarArchiveEntry mod time

…s are not reproducible

Have Jib CLI return correct version (GoogleContainerTools#3150)

codecov · 2021-03-20T21:25:00Z

Codecov Report

Merging #3159 (dd70bba) into master (5c5cf90) will increase coverage by 0.16%.
The diff coverage is 100.00%.

@@             Coverage Diff              @@
##             master    #3159      +/-   ##
============================================
+ Coverage     71.09%   71.25%   +0.16%     
- Complexity     2319     2329      +10     
============================================
  Files           279      279              
  Lines          9822     9850      +28     
  Branches        992      977      -15     
============================================
+ Hits           6983     7019      +36     
  Misses         2493     2493              
+ Partials        346      338       -8

Impacted Files	Coverage Δ	Complexity Δ
...com/google/cloud/tools/jib/image/ImageTarball.java	`92.06% <100.00%> (+0.53%)`	`9.00 <1.00> (+1.00)`
...m/google/cloud/tools/jib/tar/TarStreamBuilder.java	`100.00% <100.00%> (ø)`	`9.00 <0.00> (ø)`
...oud/tools/jib/gradle/ExtraDirectoryParameters.java	`77.27% <0.00%> (-16.85%)`	`6.00% <0.00%> (+1.00%)`	⬇️
.../cloud/tools/jib/maven/JibPluginConfiguration.java	`67.60% <0.00%> (-0.29%)`	`54.00% <0.00%> (ø%)`
.../main/java/com/google/cloud/tools/jib/cli/Jar.java	`43.75% <0.00%> (ø)`	`14.00% <0.00%> (ø%)`
...ain/java/com/google/cloud/tools/jib/cli/Build.java	`18.91% <0.00%> (ø)`	`4.00% <0.00%> (ø%)`
...m/google/cloud/tools/jib/cli/CommonCliOptions.java	`96.61% <0.00%> (+0.05%)`	`41.00% <0.00%> (+1.00%)`
...d/tools/jib/gradle/ExtraDirectoriesParameters.java	`89.18% <0.00%> (+0.30%)`	`11.00% <0.00%> (ø%)`
.../google/cloud/tools/jib/cli/logging/CliLogger.java	`92.30% <0.00%> (+0.64%)`	`17.00% <0.00%> (+1.00%)`
...jib/plugins/common/JavaContainerBuilderHelper.java	`95.45% <0.00%> (+0.90%)`	`21.00% <0.00%> (+8.00%)`
... and 4 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5c5cf90...dd70bba. Read the comment docs.

loosebazooka · 2021-03-22T22:04:33Z

I think this might be working as intended. Where are you encountering this as an issue that needs to be fixed? ReproducibleLayerBuilder adds in the behavior of reproducibility.

davidtron · 2021-03-23T09:08:15Z

The problem occurs when I create a Tar file using Jib
Jib.from(RegistryImage.named(image)).containerize(Containerizer.to(TarImage.at(outputPath).named(image)));
I've put the steps to reproduce the issue #3158 and an example unit test that will fail without the proposed change to set the tar modtime to a fixed time.

My question is - should Jib produce a reproducible tar file when the input files are the same but the command has been run at a different time?

loosebazooka · 2021-03-23T14:07:37Z

Ah okay, I see.

I think this needs to be solved in ImageTarBall's calls to TarStreamBuilder (#addByteEntry and #addBlobEntry). Those methods should be allowing a timestamp parameter.

@chanseokoh does that make sense?

chanseokoh

Thanks a lot for your contribution! I think we can accept the general direction to reset timestamps. Please check out the comment below.

jib-core/src/main/java/com/google/cloud/tools/jib/tar/TarStreamBuilder.java

chanseokoh · 2021-03-23T22:07:33Z

Note to self: CHANGELOG

chanseokoh · 2021-04-05T19:44:00Z

@davidtron just checking in, do you plan to continue the work?

… contents are not reproducible" This reverts commit 562e2b8

…s are not reproducible

davidtron · 2021-04-08T08:54:19Z

@chanseokoh yes just updated the code to pull up to ImageTarball as suggested

chanseokoh

Thanks!

Can you add some tests? I think its' not difficult to do so in TarStreamBuilderTest, as I see other tests open a tar and go over TarArchiveEntry?

jib-core/src/main/java/com/google/cloud/tools/jib/image/ImageTarball.java

jib-core/src/main/java/com/google/cloud/tools/jib/tar/TarStreamBuilder.java

jib-core/src/test/java/com/google/cloud/tools/jib/tar/TarStreamBuilderTest.java

…arball.java Co-authored-by: Chanseok Oh <[email protected]>

…mBuilderTest.java Co-authored-by: Chanseok Oh <[email protected]>

…mBuilder.java Co-authored-by: Chanseok Oh <[email protected]>

…s are not reproducible

davidtron · 2021-04-09T12:40:34Z

Just updating the test, I found that when we add a TarEntry to TarStreamBuilder we don't manipulate the time, instead rely on the fact that ReproducibleLayerBuilder which delegates to UniqueTarArchiveEntries to manage the mod time.

Just wondering if there is a clear way to articulate that in TarStreamBuilderTest so others don't waste time trying to work out why it only tests the time on entries created via addByteEntry and addBlobEntry but not addTarArchiveEntry

chanseokoh

Again, thanks for your contribution!

TarStreamBuilder is completely independent of ReproducibleLayerBuilder, so no need to bring that into your thinking. It's just that addTarArchiveEntry(TarArchiveEntry) accepts a fully-built tar entry as-is (we should not manipulate the entry at all). OTOH, the other add methods internally create a new tar entry with the given properties, so it's only natural to set those properties.

So I'm not exactly sure how you're writing tests, but you can test all of them to check if they all work in the intended way. If addTarArchiveEntry(TarArchiveEntry) adds an entry whose modTime is set to, say, 2021-04-09, then you can just check if the result tar archive has the very entry with the same modTime?

jib-core/src/main/java/com/google/cloud/tools/jib/tar/TarStreamBuilder.java

…s are not reproducible

chanseokoh

Thanks for your contribution!

I actually wanted to have separate and dedicated unit tests, but given the back-and-forth, I can take care of it after merging this.

davidtron added 2 commits March 20, 2021 21:09

GoogleContainerTools#3158 - [Jib core] Tar archives with same content…

562e2b8

…s are not reproducible

Merge pull request #1 from GoogleContainerTools/master

46d56e8

Have Jib CLI return correct version (GoogleContainerTools#3150)

google-cla bot added the cla: yes label Mar 20, 2021

davidtron mentioned this pull request Mar 23, 2021

[Jib core] Tar archives with same contents are not reproducible #3158

Closed

chanseokoh reviewed Mar 23, 2021

View reviewed changes

jib-core/src/main/java/com/google/cloud/tools/jib/tar/TarStreamBuilder.java Outdated Show resolved Hide resolved

chanseokoh added the kokoro:run label Mar 23, 2021

kokoro-team removed the kokoro:run label Mar 23, 2021

davidtron added 2 commits April 8, 2021 09:09

Revert "GoogleContainerTools#3158 - [Jib core] Tar archives with same…

f4d6757

… contents are not reproducible" This reverts commit 562e2b8

GoogleContainerTools#3158 - [Jib core] Tar archives with same content…

438e2dc

…s are not reproducible

chanseokoh added the kokoro:run label Apr 8, 2021

kokoro-team removed the kokoro:run label Apr 8, 2021

chanseokoh reviewed Apr 8, 2021

View reviewed changes

davidtron and others added 7 commits April 9, 2021 09:37

Update jib-core/src/main/java/com/google/cloud/tools/jib/image/ImageT…

f1fd866

…arball.java Co-authored-by: Chanseok Oh <[email protected]>

Update jib-core/src/test/java/com/google/cloud/tools/jib/tar/TarStrea…

11e744d

…mBuilderTest.java Co-authored-by: Chanseok Oh <[email protected]>

Update jib-core/src/test/java/com/google/cloud/tools/jib/tar/TarStrea…

3df1d7e

…mBuilderTest.java Co-authored-by: Chanseok Oh <[email protected]>

Update jib-core/src/main/java/com/google/cloud/tools/jib/tar/TarStrea…

f4bc94d

…mBuilder.java Co-authored-by: Chanseok Oh <[email protected]>

Update jib-core/src/main/java/com/google/cloud/tools/jib/tar/TarStrea…

5d19b1f

…mBuilder.java Co-authored-by: Chanseok Oh <[email protected]>

GoogleContainerTools#3158 - [Jib core] Tar archives with same content…

75d321a

…s are not reproducible

GoogleContainerTools#3158 - [Jib core] Tar archives with same content…

9c1141b

…s are not reproducible

chanseokoh reviewed Apr 9, 2021

View reviewed changes

jib-core/src/main/java/com/google/cloud/tools/jib/tar/TarStreamBuilder.java Outdated Show resolved Hide resolved

davidtron added 2 commits April 10, 2021 09:00

GoogleContainerTools#3158 - [Jib core] Tar archives with same content…

b12a4aa

…s are not reproducible

GoogleContainerTools#3158 - [Jib core] Tar archives with same content…

25a5ae2

…s are not reproducible

davidtron added 3 commits April 12, 2021 20:40

GoogleContainerTools#3158 - [Jib core] Tar archives with same content…

3bc801f

…s are not reproducible

GoogleContainerTools#3158 - [Jib core] Tar archives with same content…

cc87a86

…s are not reproducible

GoogleContainerTools#3158 - [Jib core] Tar archives with same content…

dd70bba

…s are not reproducible

chanseokoh added the kokoro:run label Apr 12, 2021

kokoro-team removed the kokoro:run label Apr 12, 2021

chanseokoh approved these changes Apr 12, 2021

View reviewed changes

chanseokoh merged commit 367fa66 into GoogleContainerTools:master Apr 13, 2021

chanseokoh mentioned this pull request Apr 13, 2021

CHANGELOG for reproducible buildTar change / refactor tests #3211

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

#3158 - [Jib core] Tar archives with same contents are not reproducible #3159

#3158 - [Jib core] Tar archives with same contents are not reproducible #3159

davidtron commented Mar 20, 2021 •

edited by chanseokoh

Loading

codecov bot commented Mar 20, 2021 •

edited

Loading

loosebazooka commented Mar 22, 2021

davidtron commented Mar 23, 2021

loosebazooka commented Mar 23, 2021 •

edited

Loading

chanseokoh left a comment

chanseokoh commented Mar 23, 2021 •

edited

Loading

chanseokoh commented Apr 5, 2021

davidtron commented Apr 8, 2021

chanseokoh left a comment

davidtron commented Apr 9, 2021 •

edited

Loading

chanseokoh left a comment •

edited

Loading

chanseokoh left a comment

#3158 - [Jib core] Tar archives with same contents are not reproducible #3159

#3158 - [Jib core] Tar archives with same contents are not reproducible #3159

Conversation

davidtron commented Mar 20, 2021 • edited by chanseokoh Loading

codecov bot commented Mar 20, 2021 • edited Loading

Codecov Report

loosebazooka commented Mar 22, 2021

davidtron commented Mar 23, 2021

loosebazooka commented Mar 23, 2021 • edited Loading

chanseokoh left a comment

Choose a reason for hiding this comment

chanseokoh commented Mar 23, 2021 • edited Loading

chanseokoh commented Apr 5, 2021

davidtron commented Apr 8, 2021

chanseokoh left a comment

Choose a reason for hiding this comment

davidtron commented Apr 9, 2021 • edited Loading

chanseokoh left a comment • edited Loading

Choose a reason for hiding this comment

chanseokoh left a comment

Choose a reason for hiding this comment

davidtron commented Mar 20, 2021 •

edited by chanseokoh

Loading

codecov bot commented Mar 20, 2021 •

edited

Loading

loosebazooka commented Mar 23, 2021 •

edited

Loading

chanseokoh commented Mar 23, 2021 •

edited

Loading

davidtron commented Apr 9, 2021 •

edited

Loading

chanseokoh left a comment •

edited

Loading